Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Script that recalibrates performance benchmarks #15446

Merged
merged 2 commits into from
Dec 3, 2024
Merged

Script that recalibrates performance benchmarks #15446

merged 2 commits into from
Dec 3, 2024
+325 −96

Conversation

igor-aptos
Copy link
Contributor

Description

Move performance benchmark values into tsv files, and add script that updates them.

How Has This Been Tested?

used the script to update values based on stable runs over the holidays.

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Move Compiler
  • Other (specify)

Checklist

  • I have read and followed the CONTRIBUTING doc
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I identified and added all stakeholders and component owners affected by this change as reviewers
  • I tested both happy and unhappy path of the functionality
  • I have made corresponding changes to the documentation

@trunk-io Trunk.io
Copy link

trunk-io bot commented Dec 2, 2024 •
edited
Loading

⏱️ 4h 41m total CI duration on this PR
Slowest 15 Jobs Cumulative Duration Recent Runs
execution-performance / single-node-performance 3h 5m 🟥 🟩 🟩 🟩 
forge-e2e-test / forge 14m 🟩 
execution-performance / test-target-determinator 13m 🟩 🟩 🟩 🟩 
rust-move-tests 13m 🟩 
rust-move-tests 13m 🟩 
rust-move-tests 12m 🟩 
test-target-determinator 7m 🟩 ⬜ 🟩 
rust-cargo-deny 5m 🟩 🟩 ⬜ 🟩 
rust-doc-tests 5m 🟩 
check-dynamic-deps 5m 🟩 🟩 🟩 🟩 🟩 
check 4m 🟩 
semgrep/ci 2m 🟩 🟩 🟩 🟩 🟩 
fetch-last-released-docker-image-tag 1m 🟩 
general-lints 1m 🟩 🟩 ⬜ 🟩 
file_change_determinator 38s 🟩 🟩 🟩 🟩 

🚨 3 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
execution-performance / single-node-performance 45m 22m +109%
execution-performance / test-target-determinator 3m 4m -25%
test-target-determinator 3m 4m -25%

settings ⋅ feedback ⋅ docs ⋅ learn more about trunk.io

msmouse
msmouse approved these changes Dec 2, 2024
View reviewed changes
Copy link
Contributor

@msmouse msmouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks boss

@igor-aptos igor-aptos added CICD:run-execution-performance-test Run execution performance test CICD:run-execution-performance-full-test Run execution performance test (full version) labels Dec 2, 2024
@igor-aptos igor-aptos force-pushed the igor/calibration_script branch from 048675c to e4d29e8 Compare December 2, 2024 22:53
@igor-aptos igor-aptos enabled auto-merge (squash) December 2, 2024 22:53
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 2, 2024

✅ Forge suite compat success on 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8

Compatibility test results for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8 (PR)
1. Check liveness of validators at old version: 010570d3b7aa20889fb5ad0e5b23800aa33f5634
compatibility::simple-validator-upgrade::liveness-check : committed: 14175.06 txn/s, latency: 2427.89 ms, (p50: 1800 ms, p70: 1900, p90: 3900 ms, p99: 14200 ms), latency samples: 498400
2. Upgrading first Validator to new version: e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8
compatibility::simple-validator-upgrade::single-validator-upgrading : committed: 7455.56 txn/s, latency: 3877.70 ms, (p50: 4300 ms, p70: 4500, p90: 4600 ms, p99: 4700 ms), latency samples: 141160
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 6865.59 txn/s, latency: 4690.10 ms, (p50: 4700 ms, p70: 4800, p90: 6700 ms, p99: 6900 ms), latency samples: 229020
3. Upgrading rest of first batch to new version: e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8
compatibility::simple-validator-upgrade::half-validator-upgrading : committed: 6640.15 txn/s, latency: 4245.59 ms, (p50: 4600 ms, p70: 4800, p90: 5600 ms, p99: 5900 ms), latency samples: 127800
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 7169.50 txn/s, latency: 4520.84 ms, (p50: 4700 ms, p70: 4800, p90: 6400 ms, p99: 6900 ms), latency samples: 238800
4. upgrading second batch to new version: e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8
compatibility::simple-validator-upgrade::rest-validator-upgrading : committed: 11153.47 txn/s, latency: 2493.46 ms, (p50: 2600 ms, p70: 2900, p90: 3400 ms, p99: 3600 ms), latency samples: 192800
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 10653.02 txn/s, latency: 2894.41 ms, (p50: 2700 ms, p70: 3300, p90: 4100 ms, p99: 5300 ms), latency samples: 348680
5. check swarm health
Compatibility test for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8 passed
Test Ok

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 2, 2024

✅ Forge suite framework_upgrade success on 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8

Compatibility test results for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8 (PR)
Upgrade the nodes to version: e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1324.89 txn/s, submitted: 1327.40 txn/s, failed submission: 2.51 txn/s, expired: 2.51 txn/s, latency: 2385.52 ms, (p50: 2000 ms, p70: 2400, p90: 4400 ms, p99: 6000 ms), latency samples: 116100
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1375.97 txn/s, submitted: 1378.73 txn/s, failed submission: 2.76 txn/s, expired: 2.76 txn/s, latency: 2311.03 ms, (p50: 1800 ms, p70: 2400, p90: 4200 ms, p99: 5700 ms), latency samples: 119560
5. check swarm health
Compatibility test for 010570d3b7aa20889fb5ad0e5b23800aa33f5634 ==> e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8 passed
Upgrade the remaining nodes to version: e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1373.23 txn/s, submitted: 1376.59 txn/s, failed submission: 3.37 txn/s, expired: 3.37 txn/s, latency: 2201.21 ms, (p50: 2100 ms, p70: 2300, p90: 3400 ms, p99: 5400 ms), latency samples: 122300
Test Ok

@github-actions GitHub Actions

This comment has been minimized.

Sign in to view
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Dec 3, 2024

✅ Forge suite realistic_env_max_load success on e4d29e8afcfed4ab0fb493f8889a6832bf65f3b8

two traffics test: inner traffic : committed: 13772.65 txn/s, latency: 2884.34 ms, (p50: 2700 ms, p70: 2700, p90: 3000 ms, p99: 3600 ms), latency samples: 5237280
two traffics test : committed: 100.03 txn/s, latency: 2107.24 ms, (p50: 1900 ms, p70: 2100, p90: 2300 ms, p99: 15600 ms), latency samples: 1760
Latency breakdown for phase 0: ["MempoolToBlockCreation: max: 2.359, avg: 1.255", "ConsensusProposalToOrdered: max: 0.328, avg: 0.301", "ConsensusOrderedToCommit: max: 0.409, avg: 0.392", "ConsensusProposalToCommit: max: 0.706, avg: 0.693"]
Max non-epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 1.09s no progress at version 2080474 (avg 0.21s) [limit 15].
Max epoch-change gap was: 0 rounds at version 0 (avg 0.00) [limit 4], 15.43s no progress at version 2080472 (avg 14.23s) [limit 16].
Test Ok

@igor-aptos igor-aptos merged commit 4815045 into main Dec 3, 2024
74 of 93 checks passed
@igor-aptos igor-aptos deleted the igor/calibration_script branch December 3, 2024 06:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@georgemitenkov georgemitenkov georgemitenkov approved these changes

@msmouse msmouse msmouse approved these changes

@perryjrandall perryjrandall Awaiting requested review from perryjrandall

@sitalkedia sitalkedia Awaiting requested review from sitalkedia

Assignees
No one assigned
Labels
CICD:run-execution-performance-full-test Run execution performance test (full version) CICD:run-execution-performance-test Run execution performance test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@igor-aptos @msmouse @georgemitenkov